Human detection device equipped with light source projecting at least one dot onto living body

ABSTRACT

A system including a light source that, in operation, projects dots onto a target, the dots being formed by first light; a first photodetector that, in operation, detects second light resulting from the projection of the dots onto the target; and a circuit. The circuit, in operation, determines whether the target is a living body or not based on the second light, performs a biometric authentication of the target, and authenticates an individual, in response to a determination that the target is a living body and a result of the biometric authentication.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of U.S. patentapplication Ser. No. 15/171,624 filed on Jun. 2, 2016, which claimspriority to Japanese Patent Application JP2015113898 filed on Jun. 4,2015, the entire disclosure of each of which is incorporated herein byreference.

BACKGROUND 1. Technical Field

The present disclosure relates to human detection devices.

2. Description of the Related Art

Heartbeat, blood flow, blood pressure, and blood oxygen saturation arewidely used as basic parameters for determining the health condition ofa person. These pieces of biological information relating to blood aretypically measured by contact-type measuring instruments. Sincecontact-type measuring instruments are attached to the body of asubject, measurement, especially, long continuous measurement, sometimesincurs the subject's discomfort.

Various attempts have been made to easily measure basic biologicalinformation for determining the health condition of a person. Forexample, Japanese Unexamined Patent Application Publication No.2005-218507 discloses a method for detecting heart rate in a non-contactmanner on the basis of image information of a face or the like obtainedwith a camera. Japanese Unexamined Patent Application Publication(Translation of PCT Application) No. 2003-517342 discloses a method formeasuring, using a white light source and a laser light source, bloodoxygen saturation on the basis of a laser Doppler effect of laser lightscattered behind the surface of a living body. Japanese UnexaminedPatent Application Publication (Translation of PCT Application) No.2014-527863 discloses a method for measuring, using an ordinary colorcamera, blood oxygen saturation while removing the influence of ambientlight.

In addition, many methods for estimating a psychological change of aperson have been proposed. For example, Japanese Unexamined PatentApplication Publication Nos. 6-54836 and 2008-237244 disclose methodsfor detecting, with thermography, a decrease in temperature at a noseportion that occurs when a person feels stress (nervous) orconcentrates.

SUMMARY

In one general aspect, the techniques disclosed here feature a systemincluding a light source that, in operation, projects dots onto atarget, the dots being formed by first light; a first photodetectorthat, in operation, detects second light resulting from the projectionof the dots onto the target; and a circuit. Further, the circuit, inoperation, determines whether the target is a living body or not basedon the second light, performs a biometric authentication of the target,and authenticates an individual, in response to a determination that thetarget is a living body and a result of the biometric authentication.

It should be noted that general or specific embodiments may beimplemented as an element, a device, a system, a method, or anyselective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram for explaining a basic concept of how biologicalinformation is obtained in an embodiment of the present disclosure;

FIG. 1B is a diagram for explaining characteristics of an image of thesurface of a living body, obtained with a camera;

FIG. 2 is a diagram illustrating an absorption coefficient and ascattering coefficient of hemoglobin, melanin, and water, which are maincomponents of a living body, in a wavelength range from visible light tonear-infrared light;

FIG. 3A is a diagram illustrating a configuration of a human detectiondevice according to a first embodiment;

FIG. 3B is a diagram illustrating an example of a configuration of acamera and an example of an output image in accordance with the firstembodiment;

FIG. 3C is a block diagram illustrating a configuration of a computer inaccordance with the first embodiment;

FIG. 4A is a diagram illustrating a first example to which a humandetection method according to the first embodiment is applied;

FIG. 4B is a diagram illustrating a second example to which the humandetection method according to the first embodiment is applied;

FIG. 5 is a diagram illustrating a contrast calculation method inaccordance with the first embodiment;

FIG. 6A is a diagram illustrating an example of a configuration and anexample of biological information (heart rate) generated in accordancewith a second embodiment;

FIG. 6B is a flowchart illustrating a flow of an image processingprocess in accordance with the second embodiment;

FIG. 7 is a diagram illustrating a configuration of a human detectiondevice according to a third embodiment;

FIG. 8 is a diagram illustrating an overview of biological informationsensing using two cameras in accordance with the third embodiment;

FIG. 9 is a diagram illustrating transmittance characteristics of twobandpass filters in accordance with the third embodiment;

FIG. 10 is a diagram illustrating an example of pulse waves measuredwith a method according to the third embodiment;

FIG. 11 is a diagram illustrating results obtained by measuring bloodoxygen saturation with the method according to the third embodiment anda method of the related art;

FIG. 12 is a diagram illustrating a configuration of astereo-camera-type human detection device according to a fourthembodiment;

FIG. 13 is a diagram illustrating a configuration of a stereo-lens-typehuman detection device according to a fifth embodiment;

FIG. 14A is a diagram illustrating a result obtained by performingstress sensing using the human detection device according to the fifthembodiment;

FIG. 14B is a diagram illustrating a nose portion and a cheek portion inan image in accordance with the fifth embodiment;

FIG. 14C is a diagram illustrating a change in blood flow and a changein blood oxygen saturation that are obtained using the human detectiondevice according to the fifth embodiment;

FIG. 15 is a cross-sectional view schematically illustrating aconfiguration of a human detection device according to a sixthembodiment;

FIG. 16A is a diagram illustrating a nose portion and a forehead portionin an image in accordance with the sixth embodiment;

FIG. 16B is a diagram illustrating a change in total blood flow(oxyhemoglobin and deoxyhemoglobin) in time and a change inoxyhemoglobin blood flow (oxygen saturation) in time when an emotioncausing laughter is induced in accordance with the sixth embodiment;

FIG. 17A is a diagram schematically illustrating a configuration of ahuman detection device according to a seventh embodiment;

FIG. 17B is a diagram illustrating a plurality of color filters inaccordance with the seventh embodiment;

FIG. 17C is a diagram illustrating an example of images generated inaccordance with the seventh embodiment;

FIG. 18A is a diagram illustrating a configuration of a human detectiondevice according to an eighth embodiment;

FIG. 18B is a diagram illustrating a plurality of color filters inaccordance with the eighth embodiment;

FIG. 18C is a diagram illustrating an example of images generated inaccordance with the eighth embodiment; and

FIG. 18D is a diagram illustrating an example of a configuration of amulti-spectral sensor including four types of color filters of R, G, B,and IR.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

Prior to description of embodiments of the present disclosure,underlying knowledge forming the basis of the present disclosure will bedescribed.

As described above, various attempts have been made to measure basicbiological information for determining the health condition of a person.For example, a method for detecting heart rate in a non-contact manneron the basis of image information of a face or the like obtained with acamera has been proposed in Japanese Unexamined Patent ApplicationPublication No. 2005-218507. In the method according to JapaneseUnexamined Patent Application Publication No. 2005-218507, heart rate isdetermined by analyzing a spatial frequency component of an obtainedcolor image. However, since the accuracy achieved by this methoddecreases due to the influence of disturbance light, such as lightilluminating a room, stable detection is difficult.

Pulse oximeters are commonly used to measure blood oxygen saturation.Pulse oximeters radiate two wavelengths of light in a red tonear-infrared wavelength range onto a finger inserted therein andmeasure transmittance of the light. In this way, pulse oximeters areable determine a ratio between an oxyhemoglobin concentration and adeoxyhemoglobin concentration in blood. Pulse oximeters are capable ofmeasuring blood oxygen saturation with a simple configuration. However,since pulse oximeters are contact-type devices, they may make peoplefeel restrained.

Japanese Unexamined Patent Application Publication (Translation of PCTApplication) No. 2003-517342 discloses an example of a non-contact-typeblood oxygen saturation measuring device. This device measures, using awhite light source and a laser light source, blood oxygen saturation onthe basis of a laser Doppler effect of laser light scattered behind thesurface of a living body. This method, however, makes a configuration ofthe device complicated, and the resulting signal is weak.

In Japanese Unexamined Patent Application Publication (Translation ofPCT Application) No. 2014-527863, a method for measuring, using anordinary color camera, blood oxygen saturation while removing theinfluence of ambient light has been proposed. Since this method isgreatly influenced by reflected light reflected by the surface of skin,it is difficult to measure blood oxygen saturation stably at a highaccuracy.

As described above, non-contact-type blood oxygen saturation measuringmethods of the related art have issues related to the accuracy and thestability. There is substantially no non-contact-type blood oxygensaturation measuring device that is put into practical use at present.

On the other hand, many methods for estimating, using thermography, apsychological change of a person have been proposed (for example,Japanese Unexamined Patent Application Publication Nos. 6-54836 and2008-237244). In these methods, a decrease in temperature at a noseportion is detected with thermography. Since lots of arteriovenousanastomosis is located at the nose portion of a person, bloodcirculation is easily disturbed by the influence of the autonomicnervous system. A psychological change, such as stress or nervousness,influences the autonomic nervous system. The influenced autonomicnervous system causes a decrease in blood flow at the nose portion,which then causes a decrease in temperature at the nose portion. Thedevices disclosed in Japanese Unexamined Patent Application PublicationNos. 6-54836 and 2008-237244 detect such a change in temperature withthermography, thereby estimating a psychological change of the subject.

Methods using thermography have a low responsivity because it takes timefor temperature to decrease and are influenced by environmentaltemperature. It is considered that a psychological change estimatingmethod that has a high responsivity and is more resistant to theinfluence of environmental temperature can be established if blood flowat the surface of the face can be accurately measured. However, noreliable method for measuring blood flow under skin at a high accuracyin a non-contact manner is established at present. Thus, methods havingthe above-described issues and using an expensive device such asthermography are predominant.

The inventor has focused on the above-described issues and has studied aconfiguration for addressing the issues. The inventor consequently hasfound out that the issues can be addressed by obtaining an image using alight source that projects at least one dot onto the surface of a livingbody and by separating, through signal processing, a component relatingto directly reflected light in the image from a component relating todiffused light caused in the living body. A human detection deviceaccording to an aspect of the present disclosure includes at least onelight source that, in operation, projects at least one dot formed bylight onto a target including a living body, an image capturing systemthat includes photodetector cells and that, in operation, generates andoutputs an image signal denoting an image of the target on which the atleast one dot is projected, and an arithmetic circuit that is connectedto the image capturing system and, in operation, processes the imagesignal output from the image capturing system. The arithmetic circuitmay, in operation, generate and output information concerning the livingbody by using the image signal. Such a configuration allows biologicalinformation to be obtained at a high accuracy.

Principle

A principle allowing a human detection device to obtain biologicalinformation at a high accuracy will be described below.

FIG. 1A is a diagram illustrating a schematic configuration of a humandetection device according to an illustrative embodiment of the presentdisclosure. The human detection device includes a light source 1 and acamera 2, which is an image capturing system. The light source 1 is anarray point light source that projects a plurality of discretelyarranged points (also referred to as “arrayed points” or “dot pattern”herein) onto a target including a living body 3. The light source 1 isarranged such that a plurality of points are projected onto the livingbody 3. The camera 2 includes an image sensor, captures an image of aliving-body surface 4, and generates and outputs an image signal.

FIG. 1B is a diagram for explaining characteristics of the image of theliving-body surface 4, obtained by the camera 2. Outgoing light L0 fromthe light source 1 is reflected by the living-body surface 4. Surfacereflected light L1 reflected by the living-body surface 4 maintains animage of the arrayed points formed by the light source 1. In contrast,inside-body scattered light L2 that exits from the living-body surface 4after entering the living body 3 and being scattered inside the livingbody 3 no longer maintains the image of the arrayed points formed by thelight source 1 because of strong scattering caused inside the livingbody 3. The use of the light source 1 allows the surface reflected lightL1 and the inside-body scattered light L2 to be spatially separated fromeach other easily.

The living body 3 illustrated in FIG. 1A represents human skin andincludes epidermis 33, dermis 34, and a subcutaneous tissue 35. No bloodvessel is located at the epidermis 33, whereas a capillary 31 and anarteriole/venule 32 are located at the dermis 34. Since there is noblood vessel at the epidermis 33, the surface reflected light L1 doesnot contain information relating to blood (hereinafter, referred to asblood-related information). Since the epidermis 33 includes melanin thatstrongly absorbs light, the surface reflected light L1 reflected by theepidermis 33 becomes noise when blood-related information is obtained.Thus, the surface reflected light L1 is not only useless to obtainblood-related information but also disturbs acquisition of accurateblood-related information. Biological information can be detected at ahigh accuracy by suppressing the influence of the surface reflectedlight and by efficiently obtaining information of the inside-bodyscattered light.

To address the issues described above, embodiments of the presentdisclosure has a novel configuration with which directly reflected lightand inside-body scattered light are spatially separated using an arraypoint light source and an image capturing system. With this novelconfiguration, information concerning the living body can be measured ata high accuracy in a non-contact manner.

In the related art, methods using polarizing illumination such as theone disclosed in Japanese Unexamined Patent Application Publication No.2002-200050 have been used to separate directly reflected lightreflected by the living-body surface. In such methods using polarizingillumination, a polarizer having a polarized light transmission axisperpendicular to a polarization direction of illumination lightreflected by an image-capturing target is used. The influence of surfacereflected light can be suppressed by capturing an image with a camerathrough such a polarizer. However, since the degree of polarization ofsurface reflected light reflected by an uneven surface such as skinchanges depending on the position, separation of such directly reflectedlight is not sufficient. With a method according to an aspect of thepresent disclosure, the influence of surface reflected light can beeffectively suppressed since directly reflected light and scatteredlight is successfully spatially separated.

In the human detection device according to an embodiment of the presentdisclosure, the wavelength of light emitted by the light source can beset to be, for example, longer than or equal to approximately 650 nm andshorter than or equal to approximately 950 nm. This wavelength range iswithin a wavelength range of red to near-infrared light. Herein, theterm “light” is used not only for visible light but also for infrared.The above wavelength range is called “optical tissue window” and isknown as a range in which absorbance in the body is low.

FIG. 2 is a diagram illustrating wavelength dependency of a lightabsorption coefficient and an inside-body light scattering coefficientfor oxyhemoglobin, deoxyhemoglobin, melanin, and water. Light isabsorbed mainly by blood (i.e. hemoglobin) in a visible light range of650 nm or shorter, whereas light is absorbed mainly by water in awavelength range longer than 950 nm. Therefore, light in thesewavelength ranges is not suitable for obtaining biological information.In contrast, in a wavelength range from approximately 650 nm toapproximately 950 nm, the absorption coefficients for hemoglobin andwater are relatively low, and the scattering coefficient is relativelyhigh. Therefore, light of this wavelength range returns to the bodysurface after entering the body and being strongly scattered.Accordingly, with light of this wavelength range, biological informationcan be efficiently obtained.

A human detection device according to an embodiment of the presentdisclosure mainly utilizes light of this wavelength range correspondingto the “optical tissue window”. With this configuration, since the humandetection device is able to separate and detect light directly reflectedby the living-body surface and returning light that has been scatteredinside the living body, it can efficiently obtain biologicalinformation.

The present disclosure includes, for example, aspects recited in thefollowing items.

[Item 1] A human detection device includes:

-   -   at least one light source that, in operation, projects, onto a        target, at least one dot formed by first light, the target        including a person and an object other than the person;    -   an image capturing system including photodetector cells that        detect second light from the target on which the at least one        dot is projected, the image capturing system, in operation,        generating and outputting an image signal denoting an image of        the target on which the at least one dot is projected; and    -   an arithmetic circuit that is connected to the image capturing        system and that, in operation, generates and outputs information        indicating whether the person is located at a position        corresponding to each pixel included in the image denoted by the        image signal.

In the human detection device according to Item 1, the at least one dotmay comprise dots formed by light.

In the human detection device according to Item 1, the at least one dotmay comprise dots formed by light, and the dots may be arranged in aline.

In the human detection device according to Item 1, the at least one dotmay comprise dots formed by light, and the dots may be arranged in anarray.

[Item 2] In the human detection device according to Item 1, thearithmetic circuit may, in operation, derive, by using the image signal,a ratio between directly reflected light that is reflected by a surfaceof the target and scattered light that exits from inside of the targetafter entering the inside of the target and being scattered, anddetermine, based on the ratio, whether the person is located at theposition corresponding to each pixel included in the image.

[Item 3] In the human detection device according to Item 1 or 2, thefirst light may include light of a wavelength longer than or equal to650 nm and shorter than or equal to 950 nm.

[Item 4] In the human detection device according to any one of Items 1to 3, the arithmetic circuit may further, in operation, generate, byusing the image signal, information concerning the person, and outputthe generated information.

[Item 5] In the human detection device according to Item 4, theinformation concerning the person may include at least one item selectedfrom the group consisting of a heart rate of the person, a bloodpressure of the person, a blood flow of the person, and a blood oxygensaturation of the person.

[Item 6] In the human detection device according to any one of Items 1to 5, the image capturing system may further include

-   -   a bandpass filter that, in operation, passes light in a        wavelength range including at least part of a wavelength range        of the first light emitted by the at least one light source, and    -   an image sensor having an imaging surface at which the        photodetector cells are disposed and onto which light that has        passed the bandpass filter is incident.

[Item 7] In the human detection device according to any one of Items 1to 6, the arithmetic circuit may, in operation, determine, for eachpixel included in the image, whether the person is located at theposition corresponding to the pixel, based on a ratio between a standarddeviation and an average of pixel values of the pixel and pixelsneighboring the pixel.

[Item 8] In the human detection device according to any one of Items 1to 3, the arithmetic circuit may further, in operation, generateinformation indicating at least one item selected from the groupconsisting of a heart rate of the person, a blood pressure of theperson, and a blood flow of the person, based on a change in a signalover time, the signal being obtained by performing a lowpass filteringprocess on at least part of the image signal, and output the generatedinformation.

[Item 9] In the human detection device according to any one of Items 1to 8, the at least one light source may be a laser light source.

In the human detection device according to Item 9, the laser lightsource may project a random dot pattern.

[Item 10] In the human detection device according to any one of Items 1to 9, the image capturing system may further include

-   -   an image sensor having an imaging surface at which the        photodetector cells are disposed;    -   an optical system that, in operation, forms the image on the        imaging surface; and    -   an adjustment mechanism that, in operation, adjusts focus of the        optical system,    -   wherein the adjusting mechanism adjusts the focus to maximize        contrast of the image.

[Item 11] In the human detection device according to Item 4, thearithmetic circuit may, in operation, perform a face recognition processby using the image signal, and generate the information concerning theperson in a case where the image includes at least one portion selectedfrom the group consisting of a forehead, a nose, a mouth, an eyebrow,and hair of the person.

[Item 12] In the human detection device according to Item 4, theinformation concerning the person may be information concerning at leastone item selected from the group consisting of a melanin concentration,presence or absence of a spot, and presence or absence of a bruise.

[Item 13] In the human detection device according to Item 1,

-   -   the at least one light source includes        -   a first light source that projects, onto the target, a first            dot pattern formed by first light including light of a            wavelength longer than or equal to 650 nm and shorter than            or equal to 800 nm, and        -   a second light source that projects, onto the target, a            second dot pattern formed by second light including light of            a wavelength longer than 800 nm and shorter than or equal to            950 nm, and    -   the photodetector cells include        -   first photodetector cells that detect third light from the            target on which the first dot pattern is projected, and        -   second photodetector cells that detect fourth light from the            target on which the second dot pattern is projected.

[Item 14] In the human detection device according to Item 13, the imagecapturing system may further include

-   -   an image sensor having an imaging surface divided into a first        region in which the first photodetector cells are disposed and a        second region in which the second photodetector cells are        disposed,    -   a first optical system that forms the first image at the first        region, and    -   a second optical system that forms the second image at the        second region.

[Item 15] In the human detection device according to Item 14, the imagecapturing system may further include

-   -   a first bandpass filter that is disposed on a path of the third        light incident onto the first region and that passes the third        light, and    -   a second bandpass filter that is disposed on a path of the        fourth light incident onto the second region and that passes the        fourth light.

[Item 16] In the human detection device according to Item 13, the imagecapturing system may further include

-   -   an image sensor        -   having an imaging surface at which the first photodetector            cells and the second photodetector cells are disposed, and        -   including first bandpass filters that face the first            photodetector cells and pass the third light and second            bandpass filters that face the second photodetector cells            and pass the fourth light, and    -   an optical system that forms the first image and the second        image on the imaging surface.

[Item 17] In the human detection device according to Item 13, the imagecapturing system may further include

-   -   an image sensor        -   having an imaging surface at which the first photodetector            cells, the second photodetector cells, and third            photodetector cells are disposed, and        -   including first bandpass filters that face the first            photodetector cells and pass the third light, second            bandpass filters that face the second photodetector cells            and pass the fourth light, and third bandpass filters that            face the third photodetector cells and pass visible light,            and    -   an optical system that forms the first image and the second        image on the imaging surface,    -   wherein the third bandpass filters include color filters having        different transmission wavelength ranges, and    -   wherein the image sensor generates and outputs a color image        signal by using the third photodetector cells.

[Item 18] In the human detection device according to any one of Items 13to 17, the arithmetic circuit may generate information indicating ablood oxygen saturation of the person by using the first image signaland the second image signal and output the generated information.

[Item 19] In the human detection device according to any one of Items 13to 17, the arithmetic circuit may

-   -   calculate, based on the first image signal and the second image        signal, a blood flow and a blood oxygen saturation of the        person,    -   generate, based on the blood flow and the blood oxygen        saturation, information indicating at least one item selected        from the group consisting of a physical condition, an emotion,        and a concentration degree of the person, and output the        generated information.

[Item 20] In the human detection device according to any one of Items 13to 17, when the first image and the second image include at least oneportion selected from the group consisting of a forehead portion and anose portion of the person,

the arithmetic circuit may

-   -   calculate, based on the first image signal and the second image        signal, a blood flow and a blood oxygen saturation at the at        least one portion selected from the group consisting of the        forehead portion and the nose portion,    -   generate, based on a change in the blood flow over time and a        change in the blood oxygen saturation over time, information        indicating at least one item selected from the group consisting        of a physical condition, an emotion, and a concentration degree        of the person, and    -   output the generated information.

[Item 21] In the human detection device according to any one of Items 13to 17, when the first image and the second image include a foreheadportion and a nose portion of the person,

the arithmetic circuit may

-   -   calculate, based on the first image signal and the second image        signal, a blood flow and a blood oxygen saturation at the        forehead portion and a blood flow and a blood oxygen saturation        at the nose portion,    -   generate, based on comparison of a change in the blood flow over        time and a change in the blood oxygen saturation over time at        the forehead portion with a change in the blood flow over time        and a change in the blood oxygen saturation over time at the        nose portion, information indicating at least one item selected        from the group consisting of a physical condition, an emotion,        and a concentration degree of the person, and    -   output the generated information.

Embodiments of the present disclosure will be described in more detailbelow. The following embodiments relate mainly to a human detectiondevice that measures biological information in a non-contact manner,assuming that a face of a person is a living-body surface. Note thattechniques of the embodiments of the present disclosure are applicablenot only to a face of a person but also to portions other than the faceof a person or to skin of animals other than human.

First Embodiment

A system in which a technique of the present disclosure is applied tohuman detection will be described as a first embodiment. Developmentrelating to human detection is underway for the purpose of detectingdisaster victims buried under rubble or the like at a disaster site, forexample. Finding disaster victims within 72 hours from occurrence of adisaster is critical in terms of the survival rate of disaster victims.Accordingly, a simple and stable human detection system is needed. Thehuman detection technology is also utilized in fields of security andtransportation. The human detection technology plays an important roleto find an intruder in the field of security and to detect footpassengers in the field of transportation. There is also an increasingneed for a system capable of selectively detecting a living body(especially, human body) in an image including various constructions orobjects.

FIG. 3A is a diagram illustrating a schematic configuration of a humandetection system according to the first embodiment. As illustrated inFIG. 3A, the human detection system according to the first embodimentincludes a light source 1, an image capturing system (i.e., imagecapturing device or camera) 2, and a computer 20. The light source 1 islocated at a position separate from a living body 3, for example, ahuman body and emits a light beam of a near-infrared wavelength range.The image capturing system 2 is capable of recording an image of aliving-body surface 4 irradiated with light. The computer 20 separatesand measures a component relating to directly reflected light reflectedby the living-body surface 4 and a component relating to scattered lightcaused inside the body by using the captured image and calculatesbiological information on the basis of an intensity of the directlyreflected light and an intensity of the scattered light, and outputs theresulting biological information.

The light source 1 is designed to project a dot pattern onto theliving-body surface 4. Typically, a dot pattern is a collection oftwo-dimensionally arranged small bright points. A one-dimensionallyarranged dot pattern may be used depending on the usage. In the firstembodiment, for example, a random dot pattern laser projector RPP017ESavailable from Osela Inc. in Canada is used as the light source 1. Thislaser light source emits a near-infrared laser beam of 830 nm andprojects a laser dot pattern including 57446 points in a 45°×45° viewingangle.

FIG. 3B is a diagram illustrating an example of a configuration of theimage capturing system (hereinafter, also referred to as a camera) 2 andan example of a generated image. The camera 2 includes a lens 5 and acamera casing 6. The lens 5 may be a set of a plurality of lenses. Thecamera casing 6 includes therein an image sensor 7 and a bandpass filter8 that passes only light of a wavelength of 830 nm±10 nm, which is thewavelength for the light source 1. The image sensor 7 has an imagingsurface at which photodetector cells are disposed.

In the case where the subject is a person, the image sensor 7 obtains animage including a plurality of points each having a brightnesscorresponding to an infrared reflectance at a corresponding position, asillustrated at the center in FIG. 3B. An arithmetic circuit 22 includedin the computer 20 is able to detect only a living body from this imageby performing image signal processing, as illustrated on the right inFIG. 3B. As described before, a living body has a specific opticalproperty called “optical tissue window” for a wavelength range of red tonear-infrared light. Since human skin has a small absorption coefficientand a large scattering coefficient in this wavelength range, light thathas transmitted through the skin surface, which is the living-bodysurface 4, repeats multiple scattering inside the body to scatter andthen exits from a wide area of the living-body surface 4. The livingbody characteristically has a high proportion of scattered lightrelative to directly reflected light in the above wavelength range. Incontrast, objects other than the living body has a high proportion ofdirectly reflected light relative to scattered light. Accordingly, aliving body can be detected based on the ratio between scattered lightand directly reflected light.

FIG. 3C is a block diagram illustrating a configuration of the computer20. The computer 20 includes an input interface (IF) 21, the arithmeticcircuit 22, a memory 24, a control circuit 25, an output interface (IF)23, and a display 26. The input IF 21 is electrically connected to thecamera 2. The arithmetic circuit 22 performs the aforementioned signalprocessing. The memory 24 stores various kinds of data. The controlcircuit 25 controls operations of the entire device. The output IF 23outputs data. The display 26 displays a processing result. Thearithmetic circuit 22 may be an image processing circuit, for example, adigital signal processor (DSP). The control circuit 25 may be anintegrated circuit, for example, a central processing unit (CPU) or amicrocomputer. The control circuit 25 runs a control program stored, forexample, in the memory 24 to perform control, such as providing aninstruction to switch on to the light source 1, an instruction tocapture an image to the camera 2, and an instruction to performcomputation to the arithmetic circuit 22. The control circuit 25 and thearithmetic circuit 22 may be implemented by a single circuit. In thisexample, the computer 20 includes the display 26; however, the display26 may be an external device electrically connected to the computer 20wirelessly or by a cable. The computer 20 may obtain, via acommunication circuit (not illustrated), image information from thecamera 2 located at a remote place.

An example of the human detection method that is carried out usingactual data will be described below.

FIG. 4A illustrates an example of an image obtained by an ordinarycamera that detects visible light. The central part shows a face F of aperson. An image on the left in FIG. 4B is an image that is obtained forthe same scene as that of FIG. 4A by the camera 2 according to the firstembodiment, with the place illuminated with the light source 1 of awavelength of 830 nm. In this image, it is difficult to recognize theface F due to strong reflection by a box B located in the foreground.Accordingly, to detect a living body, the arithmetic circuit 22calculates contrast of directly reflected light and scattered light froma near-infrared image.

FIG. 5 is a diagram illustrating an example of a pixel region used tocalculate contrast. Image data is stored as two-dimensional intensitydata in the memory 24. Here, Pij denotes data of a pixel located at ani-th column in the horizontal (x) direction and a j-th row in thevertical (y) direction. Contrast Cij for this pixel (i, j) is defined asfollows:

Cij=Sij/Aij.

Here, Sij and Aij respectively denote a standard deviation and anaverage of pieces of data of pixels in a 7×7 pixel region centered atthe pixel (i, j) and expressed by the corresponding equationsillustrated in FIG. 5. Since the standard deviation Sij decreases as theratio of scattered light to directly reflected light increases, thevalue of the contrast Cij decreases. After repeatedly performing thisprocessing for all pixels, the arithmetic circuit 22 extracts onlypixels for which the value of the contrast Cij is within a predeterminedrange. An example of an image that shows a part of a region where0.2<Cij<0.47 is an image on the right in FIG. 4B. In this image, pixelsfor which the value of the contrast Cij is within the above range areshown in white, and the rest of the pixels are shown in black. The imageindicates that a living body (i.e., the face F) is correctly extracted.

As described above, the arithmetic circuit 22 according to the firstembodiment calculates the contrast Cij, which is a ratio between thestandard deviation and the average of pixel values of a specific pixelincluded in an image and a plurality of pixels neighboring the specificpixel. Based on the value of the contrast Cij, the arithmetic circuit 22is able to determine whether a living body is located at a positioncorresponding to the specific pixel and output information indicatingthe presence or absence of the living body.

According to the first embodiment, a living body hidden behind manyobjects can be efficiently detected by utilizing a specific opticalproperty of a living body. The average and the standard deviation arederived for a 7×7 pixel region to derive a contrast of the image (i.e.,contrast of directly reflected light and scattered light) in thisexample; however, this size of the pixel region is merely an example.The size (i.e., the number of pixels) of the pixel region used tocompute the contrast may be appropriately set in accordance with thedensity of a plurality of dots formed by the light source 1 and theresolution of the camera 2. To suppress variance in the calculationresult, a plurality of (e.g., three or more) points may be included in apixel region subjected to computation. The accuracy of the calculatedcontrast value improves by increasing the number of pixels included inthe region subjected to computation; however, the resolution of theresulting image of the living body decreases. Accordingly, the number ofpixels included in the region subjected to computation may beappropriately set in accordance with the configuration and usage of thesystem. Likewise, the predetermined contrast range is not limited to0.2<Cij<0.47 and is appropriately set in accordance with theconfiguration and usage of the system.

The system capable of detecting a living body in real time from acaptured image as in the first embodiment is applicable to varioususages. Typical usage examples will be described below.

(1) Finding Disaster Victims at Time of Disaster

Quickly finding disaster victims buried in rubble in response tooccurrence of a natural disaster, such as earthquake, tsunami, or debrisflow, is particularly important to save people's lives. There is “goldentime of 72 hours”, which indicates that the survival rate greatlydecreases after 3 days passes, and it is necessary to quickly finddisaster victims in chaos. The use of the system according to the firstembodiment makes it possible to detect disaster victims hidden behindrubble in real time by capturing an image even in a circumstance whererubble is scattered everywhere. Since the system is small, the systemcan be installed on a drone, for example. This configuration makes itpossible to capture an image while remotely controlling the system at aremote location and to search for survivors even if a disaster site isdifficult to access because of a risk of a secondary disaster.

(2) Monitoring

Surveillance cameras are widely used and contribute to safe and securedaily life. As the number of surveillance cameras increases, it becomesmore important how and who checks a video image captured by thesurveillance cameras. Since it is difficult for a person to check theimage all the time, the image is often used such that the image isaccumulated and the image is checked after occurrence of a problem(crime) to grasp the situation. A utilization method may be adopted inwhich the moment at which a problem occurs is captured from a real-timeimage and the problem is immediately handled. The use of the techniqueaccording to the first embodiment of the present disclosure makes itpossible to construct a system that recognizes a person when the personenters the field of view of a surveillance camera and warns a person incharge to prompt the person in charge to check the image in real time. Asystem can be constructed that frees the person in charge from thenecessity of standing by in front of the monitor of the surveillancecamera and that displays a warning and the image on a mobile terminalcarried by the person in charge upon detecting a person. Such a systemis suitable for monitoring at a backdoor of a warehouse or buildingwhere people rarely appear or a place where access is restricted. Inaddition, for a place, such as a building, where careful monitoring isperformed with many surveillance cameras, highlighting a video image inwhich a certain person is detected may be useful to prevent an unusualsituation from being overlooked or to find out an unusual situation atan early stage.

As for monitoring, development of a method in which a computer performsobject recognition is in progress thanks to the advance in the imagerecognition technology, instead of a traditional method in which aperson judges a monitoring image. For such a usage, a common method isthat an image is transmitted to a host computer and the host computerperforms recognition. However, since this method requires image data betransmitted to the host computer, this method involves issues such as anincreasing amount of communications, a decreasing communication rate,and an increasing load of the host computer. If a surveillance camera iscapable of performing preliminary recognition and judgement on an image,the load for communication, storage, and computation can be greatlyreduced. However, if the recognition is not sufficiently reliable, therecognition may lead to overlooking of an event. Since a person can behighly reliably detected with the human detection method according tothe first embodiment, only a partial image including a person can beselectively transmitted to the host computer upon detecting the person.Consequently, the surveillance system can be efficiently operated.

In addition, the progress in the image recognition technology makes itpossible to identify an individual from an image at a high accuracy. Interms of identification of an individual from an image, a method inwhich an image is transmitted to a host computer and the host computerperforms recognition is commonly used; however, this method alsoinvolves issues relating to load for communication, storage, andcomputation as described above. Specifically, an operation forextracting a face portion for face recognition imposes a heavy loadduring computation. The use of the human detection method according tothe first embodiment allows the face portion to be easily extracted froman image. Accordingly, only the part of the image for the face portioncan be transmitted to the host computer for individual identification,and the load for identifying an individual can be greatly reduced.Further, if the number of people to be identified is limited, asurveillance camera is able to immediately identify an individualwithout using the host computer by registering characteristics of thepeople in the surveillance camera in advance.

(3) Vehicles

Installation of the system according to the first embodiment in avehicle makes it possible to recognize foot passengers on the street andimplement safer driving. Even when a person is hidden behind an objectand is scarcely seen, the system can detect the person and warn thedriver. In terms of automated driving, in a situation where a vehiclecannot stop in response to breaking and an accident is inevitable evenif the vehicle changes the direction to the left and to the right, aquestion about which direction the vehicle should head occurs. In such acase, it is effective to detect living bodies with the system accordingto the first embodiment and to change the heading direction to adirection in which the vehicle can avoid people. Since it is desiredthat the system quickly and highly accurately detect living bodies insuch a usage, the system according to the first embodiment isparticularly suitable.

(4) Switching Based on Detection of Person

There is a wide variety of usages in which power is switched on and offby detecting a person. For example, there are usages in which switchingof a device such as an air-conditioner or a light is controlled bydetecting a person in a room, an automatic door is controlled at a highaccuracy, a traffic light for foot passengers is controlled by detectingof a foot passenger at crossing, and brightness of an illumination of avending machine is changed. The first embodiment is applicable to suchusages. The use of the system according to the first embodiment canimplement a sophisticated switch that does not respond to an object orpet but responds only to people. In such a usage, a small humandetection sensor unit including the light source, the camera (imagecapturing system), and the signal processing device (arithmetic circuit)of the system according to the first embodiment may be constructed.

(5) Biometric Authentication

Biometric authentication, such as fingerprint authentication, irisauthentication, and vein authentication, is widely used as a method forauthenticating an individual. With the increasing use of suchauthentication, cases and a risk of spoofing in biometric authenticationare increasing. An image duplication technology, such as copying, hasbeen used in image-based authentication. Recently, with an increasinguse of iris authentication and a three-dimensional printer, a risk ofspoofing using a highly precise duplicate is increasing. As acountermeasure for such a risk, a two-step authentication system iseffective. For example, a method is effective in which ordinarybiometric authentication is performed after checking that a target is aliving body using the human detection system according to the firstembodiment. By checking that the target is a living body using the humandetection system according to the first embodiment, the reliability ofbiometric authentication can be increased.

Second Embodiment

A system in which a technique of the present disclosure is applied tobiological information sensing will be described as a second embodiment.With a growing interest in healthcare, importance of constant biologicalinformation sensing is increasing. A system capable of measuringbiological information in a non-contact manner at all times is essentialnot only at hospitals but also for health management in daily life.

The system according to the second embodiment is capable of monitoringheart rate and a fluctuation of heartbeat in a non-contact manner. Thesystem has a physical configuration substantially the same as that ofthe first embodiment illustrated in FIG. 1. A difference from the firstembodiment is a method of signal processing performed by the arithmeticcircuit 22. Referring to FIGS. 6A and 6B, signal processing inaccordance with the second embodiment will be described below.

FIG. 6A is a diagram illustrating an overview of signal processing inaccordance with the second embodiment. FIG. 6B is a flowchartillustrating a flow of the signal processing. The arithmetic circuit 22performs a known face recognition process on a video image obtained bythe image sensor 7 and extracts a region for a specific portion (e.g.,forehead portion) of the face (step S101). An image on the right end inFIG. 6A illustrates an example of values (pixel values) of pixelsincluded in the extracted region. The arithmetic circuit 22 performs atwo-dimensional lowpass filtering process on at least part of data(i.e., image signal) of the pixel values included in the region toremove a directly reflected light component (i.e., high-frequencycomponent) (step S102). Then, the arithmetic circuit 22 calculates anaverage of the pixel values (i.e., reflected light intensities) in theregion (step S103). The average of the reflected light intensitiesfluctuates over time as illustrated in a graph at a lower part of FIG.6A. Since arterial blood ejected from the heart moves through a bloodvessel while having a fluctuation called a pulse wave, an amount ofabsorbed near-infrared light changes with the pulse. Heart rate can bedetermined from a period of this fluctuation. Further, blood pressure orblood flow can be estimated from an amplitude of the pulse wave (stepS104). As described above, the arithmetic circuit 22 is able to generateinformation concerning at least one of heart rate, blood pressure, andblood flow of a living body, on the basis of a change in a signal overtime, the signal being obtained by performing a lowpass filteringprocess on at least part of an image signal.

It is known that mental stress can be estimated from a fluctuation ofheart rate over time. It is known that a heartbeat interval fluctuateswhen the autonomic nervous system is functioning normally and thatstress makes the fluctuation in heartbeat interval less frequent. Thearithmetic circuit 22 according to the second embodiment is able todetect whether mental stress is imposed or how much stress is imposed,on the basis of a change in the fluctuation of heartbeat interval. Toperform stress sensing in daily life at all times, a non-restraining andnon-contact heartbeat sensing technique such as the second embodiment isessential.

Many methods for monitoring, with an ordinary visible light camera or anear-infrared camera, heartbeat in a non-contact manner have beenproposed. Since separation of the surface reflected light component andthe scattered light component is insufficient in these methods of therelated art, non-contact measurement using such methods is easilyinfluenced by disturbance light and stable and highly accuratemeasurement is difficult. Spatially separating the surface reflectedlight component and the scattered light component as in the secondembodiment enables stable and highly accurate heartbeat measurement tobe performed.

The use of the system according to the second embodiment makes itpossible to monitor heart rate or blood pressure at all times includinga period when the subject is sleeping, without restraining the subject.Consequently, a system can be constructed which monitors the conditionof a patient at a hospital all the time and issues a warning to amedical personnel when something is wrong with the patient. At home, forexample, heart rate of a patient who has the sleep-apnea syndrome can bemonitored at nighttime. Further, since stress sensing can be performedeasily in daily life as described above, people can live a full life.

Third Embodiment

A system for measuring blood oxygen saturation in a non-contact mannerwill be described as a third embodiment. The main role of blood is toreceive oxygen at lungs, carries oxygen to tissues, receives carbondioxide from tissues, and carries carbon dioxide back to lungs.Approximately 15 g of hemoglobin is present in 100 ml of blood.Hemoglobin loaded with oxygen is called oxyhemoglobin (HbO₂), whereashemoglobin not loaded with oxygen is called hemoglobin ordeoxyhemoglobin (Hb). As illustrated in FIG. 2, oxyhemoglobin anddeoxyhemoglobin have different light absorption properties.Oxyhemoglobin absorbs infrared light of wavelengths exceedingapproximately 830 nm relatively well, whereas deoxyhemoglobin absorbsred light (e.g., a wavelength of 660 nm) relatively well. There is nodifference between absorbance of oxyhemoglobin and absorbance ofdeoxyhemoglobin for near-infrared light of a wavelength of 830 nm.Accordingly, in the third embodiment, transmitting light of twowavelengths, that is, 660 nm and 830 nm, are measured. A ratio (oxygensaturation) between two types of hemoglobin can be determined from aratio between transmitting light of infrared light and transmittinglight of red light. Oxygen saturation is a value indicating how muchhemoglobin in blood is loaded with oxygen. The oxygen saturation isdefined by Equation below.

Oxygen saturation=C(HbO₂)/[(C(HbO₂)+C(Hb)]×100(%), where C(Hb) denotes adeoxyhemoglobin concentration, and C(HbO₂) denotes an oxyhemoglobinconcentration.

A living body includes a non-blood components that absorb light of awavelength of red to near-infrared light; however, a fluctuation inlight absorbance over time results mainly from hemoglobin in arterialblood. Accordingly, a blood oxygen saturation can be measured at a highaccuracy, based on a fluctuation of absorbance. Arterial blood ejectedfrom the heart moves through a blood vessel as a pulse wave, whereasvenous blood does not have a pulse wave. Light radiating a living bodytransmits through the living body after being absorbed at each layer ofthe living body, such as arteries, veins, and non-blood tissues;however, thickness of such tissues other than arteries does not changeover time. Accordingly, scattered light from the living body shows achange in intensity over time in response to a change in the thicknessof an arterial blood layer due to the pulse. This change reflects achange in thickness of the arterial blood layer, and does not containthe influence of venous blood and tissues. Thus, by focusing only on thechange in the scattered light component, information concerning arterialblood can be obtained. Also, heart rate can be determined by measuring aperiod of a change in the component over time.

FIG. 7 is a diagram illustrating a configuration of the system accordingto the third embodiment. The system includes light sources 101 and 102,cameras 201 and 202, and a computer 20. The light sources 101 and 102are two array point light sources that are disposed at positionsseparate from a living body 3 and emit a light beam of a wavelength ofnear-infrared light (e.g. wavelength of 830 nm) and a light beam of awavelength of red light (e.g. wavelength of 660 nm), respectively. Thecameras 201 and 202 are two image capturing systems capable of recordingan image of a living-body surface 4 irradiated with light. The computer20 separates and measures an intensity of directly reflected lightreflected by the living-body surface 4 and an intensity of scatteredlight caused in the body from the obtained image and calculatesbiological information from the intensity of the directly reflectedlight and the intensity of the scattered light. In the third embodiment,the system includes the light sources 101 and 102, which are two arraypoint light sources with different wavelengths, and the cameras 201 and202 respectively corresponding to the light sources 101 and 102 in orderto measure blood oxygen saturation.

FIG. 8 is a diagram illustrating a configuration of the image capturingsystems. Each of the cameras 201 and 202, which are the image capturingsystems, includes a lens 5 and a camera casing 6. The camera casing 6 ofthe camera 201 includes therein an image sensor 7 and a bandpass filter801 that selectively passes near-infrared light (wavelength of 830 nm).The camera casing 6 of the camera 202 includes therein the image sensor7 and a bandpass filter 802 that selectively passes red light(wavelength of 660 nm).

For example, a random dot pattern laser projector RPP017ES availablefrom Osela Inc. in Canada can be used as the light source 101. Thislaser light source is a 830-nm near-infrared laser light source andprojects a laser dot pattern including 57446 points in a 45°×45° viewingangle. For example, a random dot pattern laser projector RPP016ESavailable from Osela Inc. in Canada can be used as the light source 102.This laser light source is a 660-nm red laser light source and projectsa laser dot pattern including 23880 points in a 35°×35° viewing angle.

The computer 20 controls the cameras 201 and 202 and the light sources101 and 102 so that the two cameras 201 and 202 operate together tosimultaneously capture respective images. In this way, images based onlight having two different wavelengths are generated by the cameras 201and 202 as illustrated on the right in FIG. 8, for example.

FIG. 9 is a diagram illustrating transmittance characteristics of thebandpass filters 801 and 802. The bandpass filter 801 has atransmittance characteristic that the transmission center wavelength is830 nm and the bandwidth is 10 nm. The bandpass filter 802 has atransmittance characteristic that the transmission center wavelength is660 nm and the bandwidth is 10 nm. The transmission center wavelengthsof the bandpass filters 801 and 802 respectively match central values ofwavelengths of the light sources 101 and 102. Accordingly, the camera201, which is the image capturing system, obtains an image based onlight of a wavelength of 830 nm, whereas the camera 202, which is theimage capturing system, obtains an image based on light of a wavelengthof 660 nm.

The arithmetic circuit 22 of the computer 20 first performs a known facerecognition process on a video image and extracts a region for aspecific portion (e.g., forehead portion) of the face, as in the secondembodiment. The arithmetic circuit 22 then performs a two-dimensionallowpass filtering process on data of pixels in the region to remove adirectly reflected light component. Then, the arithmetic circuit 22calculates an average of pixel values of the pixels in the region. Thearithmetic circuit 22 performs the above process for each of the camera201 for 830 nm and the camera 202 for 660 nm. The averages thus obtainedindicate intensities of scattered reflected light from the living body3.

FIG. 10 is a diagram illustrating an example of changes in the obtainedscattered reflected light intensities over time. The reflected lightintensity fluctuates over time for both near-infrared light (wavelengthof 830 nm) and red light (wavelength of 660 nm). Here, let li(830) andli(660) respectively denote an intensity of light emitted from the lightsource 101 and an intensity of light emitted from the light source 102at the living-body surface 4, and let Δl(830) and Δl(660) each denote anaverage of the fluctuation component of the scattered reflected lightfrom the living body 3. Blood oxygen saturation SpO₂ is calculated usingEquation below.

SpO₂=a+b*(log(Δl(660)/li(660)))/(log(Δl(830)/li(830))), where a and bcan be decided upon based on a relationship with measured valuesobtained by an existing pulse oximeter.

To check the accuracy of the measuring instrument, oxygen saturation ata fingertip instead of the forehead is measured using the systemaccording to the third embodiment. Specifically, oxygen saturation at afingertip is measured while blood flow is stopped by pressuring theforearm at a certain pressure (200 mmHg) using a belt (cuff) used tomeasure the blood pressure.

A commercially available pulse oximeter to which a finger is inserted isattached to the index finger. Oxygen saturation at the middle finger ismeasured in a non-contact manner using the system according to the thirdembodiment. The above values a and b are decided upon by the firstmeasurement, and blood oxygen saturation SpO₂ is measured by the secondand subsequent measurement.

FIG. 11 illustrates a comparison result of the measured values obtainedusing the pulse oximeter and measured values obtained in accordance withthe third embodiment. Since both results substantially match, FIG. 11indicates that measurement is performed accurately. In the method of thethird embodiment, not only blood oxygen saturation but also heart ratecan be simultaneously measured based on the pulse waves illustrated inFIG. 10.

It is known that stress and tiredness can be measured from thefluctuation or a frequency characteristic of the pulse wave. The use ofthe system according to the third embodiment makes it possible toestimate a mental condition, such as stress, and a physical condition ofa subject from the pulse wave in a non-contact manner.

Fourth Embodiment

A method for measuring blood oxygen saturation by using one camera willbe described as a fourth embodiment. In third embodiment, two camerasare used and signals for the light sources with different wavelengthsare obtained by the respective cameras. This method has an advantage inthat existing cameras can be used. However, since it is necessary tocapture images by controlling two cameras to operate together, theconfiguration of the system becomes complicated. Also, since theobtained data is individual pieces of video data for the two cameras,synchronized data processing becomes complicated. To avoid suchcomplexities, in the fourth embodiment, a camera capable ofsimultaneously obtaining data of images for two wavelengths isimplemented.

FIG. 12 is a diagram illustrating a configuration of a human detectiondevice according to the fourth embodiment. This human detection devicehas a structure of a twin-lens reflex camera including two imagingoptical systems 201 and 202. Accordingly, herein, such a configurationis referred to as a “stereo camera configuration”. The human detectiondevice (also referred to as a “camera”) includes a light source 101(wavelength of 830 nm), which is a first laser point light source, and alight source 102 (wavelength of 760 nm), which is a second laser pointlight source. The light emitted by the light sources 101 and 102 andreflected by a living body 3 respectively passes through bandpassfilters 801 and 802. Then, the propagating direction of the light isbent by mirrors 901 and 902 by 90 degrees, and images are formed onimaging surfaces of image sensors 701 and 702 via lenses 501 and 502,respectively. The bandpass filters 801 and 802 are narrow-band bandpassfilters that pass only light of a wavelength of 830±15 nm and light of awavelength of 760±15 nm that correspond to wavelengths of the two lightsources 101 and 102, respectively.

In response to pressing of a shutter button 11, the two light sources101 and 102 switch on, and the image sensors 701 and 702 simultaneouslyobtain images of the living body 3. The obtained images are convertedinto images of a stereo-image format by an image processing processor(corresponding to the arithmetic circuit 22 in FIG. 3C), are subjectedto image signal processing, and are accumulated in a storage device(corresponding to the memory 24 in FIG. 3C). The following processing issubstantially the same as that of the second and third embodiments.

According to the fourth embodiment, by configuring an image capturingsystem as one stereo camera, the entire system becomes compact, and theconfiguration of the following signal processing system from imagesignal processing to calculation of oxygen saturation can be simplified.In this way, a simple and high-speed operation can be implemented.

For example, 760 nm and 830 nm, which are in a near-infrared range, canbe used as wavelengths of the two light sources. Since a difference inabsorbance between oxyhemoglobin and deoxyhemoglobin is larger for 660nm used in the second and third embodiments than that for 760 nm, oxygensaturation can be measured more accurately for the wavelength of 660 nm.However, since the wavelength of 660 nm is in a visible light range, theuse of this wavelength may impose a load on the subject. Further, sincelight of a fluorescent lamp and a light-emitting diode (LED)illumination contains a component of the wavelength of 660 nm,measurement is easily affected by ambient light. In the fourthembodiment, the wavelength of 760 nm is selected in consideration ofsuch issues. Since a local absorbance peak of deoxyhemoglobin is at 760nm, it is effective to use a wavelength of 760 nm to 780 nm if thewavelength of the light source having a shorter wavelength is set in thenear-infrared range. The wavelengths used are not limited to the aboveones, and may be appropriately selected in accordance with the usage andthe use environment.

Fifth Embodiment

Another method for measuring blood oxygen saturation by using one camerawill be described as a fifth embodiment. In the fourth embodiment, thestereo camera configuration in which one camera includes two opticalsystems and two image sensors is employed. In the fifth embodiment, asystem is employed which obtains two different images corresponding totwo wavelengths with one image sensor by dividing an image using aplurality of lenses. The configuration according to the fifth embodimentis referred to as a “stereo lens configuration”. The system of thestereo lens configuration will be described with reference to FIG. 13.

FIG. 13 is a cross-sectional view schematically illustrating a part of ahuman detection device according to the fifth embodiment. In the fifthembodiment, an imaging surface of an image sensor 7 includes a firstregion 7 a in which first photodetector cells are disposed and a secondregion 7 b in which second photodetector cells are disposed. Althoughnot illustrated in FIG. 13, the human detection device includes, forexample, in a camera casing 6, two light sources that project dotpatterns formed by light of two wavelengths of 830 nm and 760 nm. Asillustrated in FIG. 13, a lens 5, which is an optical system, includestherein two sets of lenses 501 and 502. The set of lenses 501 and theset of lenses 502, which are the optical systems, are designed to formrespective images at the first region 7 a and the second region 7 b onthe imaging surface of the image sensor 7, respectively. Two narrow-bandbandpass filters 801 and 802 that pass light of 830 nm and 760 nmcorresponding to the wavelengths of the two light sources are disposedin front of the sets of lenses 501 and 502, respectively. Specifically,the bandpass filter 801 is disposed on a path of light entering thefirst region 7 a, whereas the bandpass filter 802 is disposed on a pathof light entering the second region 7 b.

With such a configuration, two images based on light of two wavelengthsfor the same time point can be obtained by using one image sensor 7. Thearithmetic circuit 22 calculates blood oxygen saturation from the twoimages using the method similar to that of the second to fourthembodiments. According to the fifth embodiment, since one image signalinclude information concerning two images corresponding to two differentwavelengths for the same time point, the arithmetic processing becomeseasier.

A result obtained by performing stress sensing by using the system ofthis stereo lens configuration will be described below. As describedabove, a method for detecting, with thermography, a decrease intemperature at a nose portion due to stress (nervousness) orconcentration has been proposed. The blood flow decreases at the noseportion due to a psychological change, and in response to the decreasein the blood flow, temperature decreases at the nose portion. A methodfor detecting such a temperature change with thermography is commonlyperformed. A change in temperature at the face is caused by a change inblood flow. If a change in blood flow can be measured at a highaccuracy, stress sensing can be implemented that is more accurate andmore responsive than in the case of measuring a change in the surfacetemperature, which occurs as a result of a change in blood flow.

FIG. 14A is a diagram illustrating a result of stress sensing performedusing the human detection device according to the fifth embodiment. Acold-water load for immersing the right hand into cold water (ice water)is imposed as stress. For comparison, a temperature change is measuredusing thermography at a nose portion and a cheek portion enclosed bydotted lines in FIG. 14B. FIG. 14A illustrates results of thismeasurement. The temperature at the nose portion gradually decreases inabout three minutes after the cold-water load is started to be imposedand becomes stable after decreasing by approximately 1.2° C. FIG. 14Aindicates that temperature returns to the original level in about threeminutes after the load is removed. In contrast, temperature at the cheekportion is hardly affected by the cold-water load and is stable.

FIG. 14C is a diagram illustrating a change in blood flow and a changein blood oxygen saturation that are obtained by the human detectiondevice according to the fifth embodiment that employs the stereo lensconfiguration. Data for regions corresponding to the nose portion andthe cheek portion, which are denoted by dotted-lines in FIG. 14B, areextracted from data of the blood flow and the oxygen saturation (SpO₂)at the face. A solid line denotes a change in the blood flow over time,whereas a dotted line denotes a change in the oxygen saturation (ΔSpO₂)over time. As illustrated in FIG. 14C, the blood flow tends to decreaseat the nose portion immediately after a cold stimulus is started, whichindicates the responsivity with respect to time is high. In contrast,the blood flow hardly changes at the check portion. A decrease in theoxygen saturation is observed at the nose portion in response to adecrease in the blood flow, whereas the oxygen saturation hardly changesat the cheek portion.

As is apparent from the results, many pieces of data can be obtained bymeasuring blood flow and oxygen saturation at different portions of theface. An emotion, a physical condition, and a concentration degree canbe detected at a high accuracy based on these pieces of data. The changein blood flow due to influence of the autonomic nervous system differsfrom portion to portion of the face. Thus, it is especially important tomeasure a change in blood flow at a specific portion by using thecamera. At that time, the accuracy of the measurement can be increasedby performing, at the same time, measurement at a portion where bloodflow hardly changes and using the result as a reference.

Sixth Embodiment

Another method for measuring blood oxygen saturation by using one camerawill be described as a sixth embodiment.

FIG. 15 is a sectional view schematically illustrating a configurationof a human detection device according to the sixth embodiment. The humandetection device includes a stereo adapter 10 attachable to an ordinarylens 5. The stereo adapter 10 is an attachment including four mirrors151, 152, 153, and 154 and two bandpass filters 801 and 802. The use ofthe stereo adapter 10 allows two images corresponding to two wavelengthsto be formed at two different regions of an imaging surface of an imagesensor 7. This configuration is referred to as a “stereo adapterconfiguration”.

In the stereo adapter configuration, two different images correspondingto two wavelengths can be obtained by one image sensor 7 by using twosets of facing mirrors. Although not illustrated in FIG. 15, two lightsources that respectively emit light of two wavelengths of 830 nm and760 nm are included in a camera casing 6. The stereo adapter 10 isattached to an end of the lens 5 of the camera. The two sets of mirrors(a set of mirrors 151 and 152 and a set of mirrors 153 and 154) bend thelight path twice to guide the light to the lens 5. The narrow-bandbandpass filters 801 and 802 that respectively pass light of thewavelengths of 830 nm and 760 nm corresponding to the wavelengths of thelight sources are disposed between the lens 5 and the mirrors 151, 152,153, and 154.

This human detection device is able to obtain images of two wavelengthsfor the same time point by using one image sensor 7. The basic conceptis the same as that of the fifth embedment. Since the stereo lensconfiguration can make the lens small, the entire system can beadvantageously made small. In contrast, with the stereo adapterconfiguration, the entire system becomes larger but a powerful cameralens can be used and the resolution can be improved. Also, lenses ofdifferent magnifications and zoom lenses can be used. The stereo adapterconfiguration advantageously increases the degree of freedom of thesystem.

A study for detecting an emotion of a person by using the humandetection device (i.e., camera) according to the sixth embodiment hasbeen carried out. As described in the fifth embodiment, a feeling oremotion such as stress of a human can be stably detected based on bloodflow. In response to a change in a feeling or emotion of a person, theautonomic nervous system becomes more active and blood flow on the skinsurface changes. As a result of this change in blood flow, facial colorchanges. People detect an emotion and a physical condition of a targetperson from a subtle change in facial color without any difficulty. Itis considered that a reason why a great doctor can diagnose a patient'sphysical condition and a cause of a diseases by just looking at thepatient's face is that such a doctor can identify a physical change fromthe subtle change in color of the patient's face. In addition, it issaid that, when a person who is good at reading the situation reads afeeling of a counterpart, a subtle change in facial color as well as asubtle change in facial expression play an important role. Further, tomake a situation natural and real in fields showing remarkableprogresses recently, such as game, animation, and computer graphics,studies for subtly changing the facial color of a human character arewidely carried out. As is apparent from these examples, the facial colorrepresents an emotion and a physical condition of a person, and anattempt to read a feeling by measuring the facial color has been studied(for example, Kuroda et al., “Analysis of facial color and skintemperature in emotional change and its synthesis of facial color”,Human interface 1(1), pp. 15-20, 1999). However, such an attempt todirectly measure an emotion from facial color is not suitable forpractical use because stable measurement is difficult. This is because achange in facial color differs from person to person, and stablemeasurement is difficult since a change in facial color is subtle and isstrongly influenced by disturbance light and a camera. A method forstably and highly accurately detecting an emotion by a measure otherthan measuring a change in facial color is desired.

It is known that facial color of a person is mainly decided by an amountof melanin contained in the skin surface (dermis) and concentrations ofhemoglobin (oxyhemoglobin and deoxyhemoglobin) in blood. Since theamount of melanin does not fluctuate in a short time (changes due to afactor such as aging or tanning), a change in emotion can be stablymeasured by measuring blood flow. In the sixth embodiment, instead ofmeasuring facial color, blood flow of oxyhemoglobin and deoxyhemoglobinthat changes facial color is directly measured to detect a change inemotion. As described in the fifth embodiment, a change in blood flowdiffers from portion to portion of the face because an impact of theinfluence of the autonomic nervous system differs from portion toportion of the face. For example, the nose portion is easily influencedby the autonomic nervous system because lots of arteriovenousanastomosis is located at the nose portion, whereas the forehead portionis hardly influenced by a skin blood vessel contraction nerve. Thearithmetic circuit 22 according to the sixth embodiment determines bloodflows at a plurality of different portions by computation and comparesthe obtained blood flows with each other, thereby being able to detect achange in emotion at a high accuracy.

Measurement of a change in blood flow in response to an emotional changewill be described below. The camera of the stereo adapter configurationillustrated in FIG. 15 is used to measure blood flow. A subject takes aseat in front of the camera, and an image of the subject's face iscaptured with the camera. Color images of the face of the subject areobtained while showing the subject a video image that induces, from thesecured state, feelings such as fear, laughter, surprise, and disgust.An emotional change is read based on a change in scene in the videoimage and the facial change in the color images, and a change in bloodflow at the time of such a change is measured. Blood flow is measured atthe nose portion and the forehead portion as indicated by dotted linesin FIG. 16A.

FIG. 16B is a diagram illustrating a change in total blood flow(oxyhemoglobin and deoxyhemoglobin) over time and a change in thepercentage of oxyhemoglobin blood flow (oxygen saturation) over timewhen an emotion causing laughter is induced. FIG. 16B indicates that thevalue of the total blood flow and the value of the blood oxygensaturation greatly change in response to the emotional change causinglaughter. Similar examinations are performed for other emotions. Thetotal blood flow and an amount of change in the blood oxygen saturationare determined by calculation for the case where other emotions, such assadness, surprise, depression, fear, disgust, anger, concentration, andhappiness, are induced. The same measurement is performed for twelvesubjects. Although there is a variation among individuals, the change inthe total blood flow and the change in the blood oxygen saturation haveshowed the similar tendencies for almost all the subjects. This resultindicates that an emotional change can be detected from at least one ofblood flow and oxygen saturation.

As illustrated in FIG. 16B, a relationship between oxygen saturation andblood flow differs from portion to portion of the face. Accordingly,highly accurate emotion sensing can be performed by determining bloodflow and oxygen saturation at a plurality of portions of the face. Inthe emotion sensing test performed in the sixth embodiment, measurementis performed at three portions, i.e., at the forehead, the cheek, andthe nose. A change in oxygen saturation and a change in blood flow inresponse to an emotional change differ among the forehead, the cheek,and the nose. Accordingly, an emotional change can be detected highlyaccurately by creating in advance a table indicating a relationshipbetween oxygen saturation and an amount of change in blood flow at eachportion and calculating a correlation with the actually measured valuesof oxygen saturation and blood flow.

Seventh Embodiment

A method for measuring blood oxygen saturation by using one camerawithout dividing an image by an optical system will be described as aseventh embodiment. In the second to sixth embodiments, the method fordividing light from two light sources corresponding to two wavelengths,performing sensing, and determining biological information, such asoxygen saturation, by computation has been described. A human detectiondevice according to the seventh embodiment obtains, with an imagesensor, two image signals for different wavelengths without dividing animage.

FIG. 17A is a diagram schematically illustrating a configuration of thehuman detection device according to the seventh embodiment. This humandetection device separate two images corresponding to two wavelengthsnot by the optical system but by an image sensor 703. Althoughillustration of point light sources is omitted in FIG. 17A, two lightsources that respectively emit light of a wavelength of 830 nm and lightof a wavelength of 760 nm are included in a camera casing 6. A bandpassfilter 8 that passes light of a wavelength longer than or equal to 730nm and shorter than or equal to 850 nm is disposed in front of a lens 5of the camera. The bandpass filter 8 cuts visible light and infraredlight of long wavelengths. Light that has passed the bandpass filter 8forms an image on the imaging surface of the image sensor 703 via thelens 5. Unlike ordinary image sensors, the image sensor 703 used in theseventh embodiment includes two kinds of bandpass filters (hereinafter,also referred to as color filters) that pass near-infrared light.

FIG. 17B is a diagram illustrating a plurality of color filters thatface a plurality of photodetector cells disposed on the imaging surfaceof the image sensor 703. The image sensor 703 includes a plurality ofcolor filters IR1 (also referred to as a first bandpass filter set) thatselectively pass light of a wavelength of 680 nm to 800 nm and aplurality of color filters IR2 (also referred to as a second bandpassfilter set) that selectively pass light of a wavelength of 800 nm orlonger. The color filters IR1 and IR2 are arranged in a checkeredpattern. The lower image in FIG. 17B is a diagram illustrating anexample of wavelength dependency of transmittance of the color filtersIR1 and IR2. The image sensor 703 detects, with a plurality ofphotodetector cells (also referred to as pixels), two images based onlight of 760 nm and 830 nm which are wavelengths of the two lightsources.

The arithmetic circuit 22 (FIG. 3C) separately reads data obtained bythe plurality of photodetector cells of the image sensor 703 for thewavelength of 760 nm and data obtained by the plurality of photodetectorcells for the wavelength of 830 nm. As illustrated in FIG. 17C, thearithmetic circuit 22 generates an image for the wavelength of 760 nmand an image for the wavelength of 830 nm by adding, to thecorresponding data, data for lacking pixels by interpolation. Sincethese two images completely coincide with each other, calculating bloodflow and oxygen saturation from these images is easier than calculationusing two different images. However, since the filtering performance ofthe filters is lower than that achieved in the case where bandpassfilters corresponding to respective light sources are used, there is aconcern about occurrence of color mixing between the light sources inthis method.

Eighth Embodiment

A human detection device capable of obtaining not only two imagescorresponding light sources with two wavelengths without dividing animage but also a color image will be described as an eighth embodiment.

FIG. 18A is a diagram illustrating a configuration of the humandetection device according to the eighth embodiment. Althoughillustration of point light sources is omitted also in FIG. 18, twolight sources that respectively emit light of a wavelength of 830 nm andlight of a wavelength of 760 nm are included in a camera casing 6. Inthe eighth embodiment, to obtain a color image, no bandpass filter isdisposed in front of a lens 5. Visible light and light emitted by theleaser light sources form images on an imaging surface of an imagesensor 704 via the lens 5. Unlike ordinary image sensors, the imagesensor 704 used in the eighth embodiment includes photodetector cellsthat obtain a color image and two kinds of photodetector cells thatobtain near-infrared images.

FIG. 18B is a diagram illustrating a plurality of bandpass filters (orcolor filters) disposed on the imaging surface of the image sensor 704.The lower image in FIG. 18B illustrates wavelength dependencies ofrelative sensitivities of pixels that face corresponding filters. Asillustrated in FIG. 18B, three types of color filters (R, G, and Bfilters) that respectively pass red light, green light, and blue light,filters IR-1 that pass light of 650 nm or longer, and filters IR-2 thatpass light of 800 nm or longer are arranged on the imaging surface. Thefilters IR-1 are referred to as a first bandpass filter set. The filtersIR-2 are referred to as a second bandpass filter set. R, G, and Bfilters are referred to as a third bandpass filter set. An array inwhich two G filters are disposed diagonally adjacent to each other and Rand B filters are disposed on the opposite diagonal side is the same asthe Bayer array of ordinary image sensors. This image sensor 704 differsfrom ordinary image sensors in that two filters IR-1 and IR-2 arearranged next to a basic unit of four filters arranged in the Bayerarray.

The filter IR1 of the seventh embodiment and the filter IR-1 of theeighth embodiment have different transmission wavelength ranges. Thefilter IR1 of the seventh embodiment is a relatively narrow-band filterthat passes light of a wavelength range from 650 nm to 800 nm. Incontrast, in the eighth embodiment, a filter that pass light of awavelength range of 650 nm or longer is used to simplify themanufacturing process of the image sensor 704. However, theconfiguration is not limited to this one, and the color filter describedin the seventh embodiment can also be used. The filter IR-1 of theeighth embodiment is sensitive to both 760 nm and 830 nm. Accordingly,the arithmetic circuit 22 calculates a signal corresponding to 760 nm bysubtracting the signal of the photodetector cells that face the filterIR-2 from the signal of the photodetector cells that face the filterIR-1 and then determines blood oxygen saturation. Consequently, an image(color image) of red, blue, and green, an image for the wavelength of760 nm, and an image for the wavelength of 830 nm are determined by theimage sensor 704 as illustrated in FIG. 18C.

In this configuration, color mixing is more likely to occur than in theseventh embodiment. However, a color image, blood flow, and blood oxygensaturation can be simultaneously obtained by the simple system using onecamera.

In the eighth embodiment, an example of a configuration of a biologicalinformation sensing camera that uses a multi-spectral sensor thatsupport five wavelengths including two wavelengths in an infrared rangeand three wavelengths (red, blue, and green) in a visible light rangehas been described. With the human-detection-type camera described inthe first embodiment, capturing of a color image and human detection canbe performed through measurement on four wavelengths including onewavelength in an infrared range and three wavelengths (red, blue, andgreen) in a visible light range. The multi-spectral sensor (illustratedin FIG. 18D, for example) having four types of color filterscorresponding to four wavelengths is usable for such a purpose. A colorfilter is disposed such that a near-infrared (IR) pixel is assigned toone pixel out of two green pixels of the Bayer array that is commonlyused in the image sensor. In the eighth embodiment, a camera for asystem that switches on a near-infrared illumination of 850 nm isassumed, and a filter that selectively passes the wavelength of 850 nmis selected as a near-infrared filter. The use of such a camera makes itpossible to use one camera system as an ordinary color camera and ahuman detection camera. Consequently, only one surveillance camera isneeded, and it is easier to extract a color image of a portion in whicha person is detected than in the case where two cameras are used. In theeighth embodiment, a color filter for 850 nm is used; however, thenear-infrared filter may be changed in accordance with the near-infraredlight source used.

Other Embodiments

While the embodiments of the present disclosure have been describedabove by way of example, the present disclosure is not limited to theabove embodiments and can be variously modified. The process describedin each of the above embodiments may be applied to other embodiments.Examples of the other embodiments will be described below.

In the embodiments above, laser light sources are used as the arraypoint light sources; however, light sources of other types may be used.For example, less costly LED light sources may be used. However, lightemitted by the LED light source has a lower straightness than thatemitted by the laser light source and is more likely to spread.Accordingly, a dedicated condensing optical system needs to be used orsome attention needs to be paid, such as restraining a distance betweenan image-capturing target and a camera. The number of array point lightsources is not limited to one or two, and three or more light sourcesmay be used.

The human detection device may include an adjustment mechanism thatadjusts focus of the optical system. Such an adjustment mechanism can beimplemented by, for example, a motor (not illustrated) and the controlcircuit 25 illustrated in FIG. 3C. Such an adjustment mechanism adjustsfocus of the optical system to maximize contrast of a dot pattern imageprojected onto a target by the light source. With this configuration,accuracy of calculation of contrast described in the first embodimentimproves.

The arithmetic circuit 22 may perform a face recognition process usingimage signals output by the image sensor and output informationconcerning a living body if the image includes at least one of aforehead, a nose, a mouth, an eyebrow, and hair. A system may beimplemented that displays an error if none of the forehead, the nose,the mouth, the eyebrow, and the hair of a living body is included in theimage.

The arithmetic circuit 22 may generate information concerning theepidermis including information concerning at least one of a melaninconcentration, presence or absence of a spot, and presence or absence ofa bruise on the basis of the image signal. As described above, theepidermis contains melanin that strongly absorbs light. A spot and abruise are caused as a result of an increase in melanin. Accordingly, amelanin concentration, a spot, and a bruise can be detected based on anintensity distribution of light from the living-body surface.

In the present disclosure, the double camera configuration using twocameras (FIG. 7), the stereo camera configuration (FIG. 12) in which onecamera includes two optical systems and two image sensors, the stereolens configuration (FIG. 13) using two sets of lenses and one imagesensor, the stereo adapter configuration (FIG. 15) using a lens adapter,one lens, and one image sensor, the configuration (FIG. 17A and FIG.18A) that divides an image using the image sensor have been described.As already described, since each configuration has advantages anddrawbacks, an appropriate configuration can be selected in accordancewith the usage.

As described above, according to the embodiments of the presentdisclosure, not only heart rate and blood flow but also blood oxygensaturation can be measured without restraining a subject and withoutplacing a detection device, such as a sensor, in contact with thesubject. An emotion or a physical condition of the subject can also beestimated from measured values of blood flow and oxygen saturation atdifferent portions of the subject.

What is claimed is:
 1. A system comprising: a light source that, inoperation, projects dots onto a target, the dots being formed by firstlight; a first photodetector that, in operation, detects second lightresulting from the projection of the dots onto the target; and acircuit, wherein the circuit, in operation, determines whether thetarget is a living body or not based on the second light, performs abiometric authentication of the target, and authenticates an individual,in response to a determination that the target is a living body and aresult of the biometric authentication.
 2. The system according to claim1, wherein the circuit, in operation, determines whether the target is aliving body or the duplicate of a living body.
 3. The system accordingto claim 1, wherein the second light includes: reflected light that isreflected by the target; and scattered light that exits from inside ofthe target after entering the inside of the target and being scattered,and the circuit, in operation, determines whether the target is a livingbody or not, based on the directly reflected light and the scatteredlight.
 4. The system according to claim 3, wherein the circuit, inoperation, determines whether the target is a living body or not, basedon a ratio between the directly reflected light and the scattered light.5. The system according to claim 1, wherein the first light has awavelength longer than or equal to 650 nm and shorter than or equal to950 nm.
 6. The system according to claim 1, wherein the light source isan array point light source that projects a plurality of discretelyarranged dot pattern onto the living body.
 7. The system according toclaim 1, wherein the biometric authentication includes at least onepiece of authentication selected from the group consisting offingerprint authentication, iris authentication, and veinauthentication.
 8. A method comprising: causing a light source toproject dots onto a target, the dots being formed by first light;causing a first photodetector to detect second light resulting from theprojection of the dots onto the target; determining whether the targetis a living body or not based on the second light; performing abiometric authentication of the target, and authenticating anindividual, in response to a determination that the target is a livingbody and a result of the biometric authentication.